Method and apparatus for image processing and computer storage medium

ABSTRACT

A method and an apparatus for processing an image are provided. The method may include: acquiring a set of image sequences, the set of image sequences including a plurality of image sequence subsets divided according to similarity measurements between image sequences, each image sequence subset including a basic image sequence and other image sequence, wherein a first similarity measurement corresponding to the basic image sequence is greater than or equal to a first similarity measurement corresponding to the other image sequence; creating an original three-dimensional model using the basic image sequence; and creating a final three-dimensional model using the other image sequence based on the original three-dimensional model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of Chinese Patent Application No.202010624149.2, titled “METHOD AND APPARATUS FOR IMAGE PROCESSING ANDCOMPUTER STORAGE MEDIUM”, filed on Jun. 30, 2020, the content of whichis incorporated herein by reference in its entirety.

TECHNICAL FIELD

Embodiments of the present disclosure relate to the field of imageprocessing, and more particularly, to a method and apparatus forprocessing an image and to a computer storage medium, which may beapplicable to the field of automatic-driving.

BACKGROUND

In the field of automatic-driving, particularly in autonomous parking orcruising scenes such as in parking lots or residential roads, it isdesirable to reconstruct and fuse a three-dimensional model for thesescenes based on driving record images acquired, for example, by avehicle or other device for acquiring these images. However, since theamount of data of the driving record images is usually very large,reconstruction and fusion of the three-dimensional model may involve alarge amount of computation and occupy a large amount of systemresources. In addition, as the amount of data of the driving recordimages is further increased, the amount of computation and occupiedsystem resource for reconstructing and fusing the three-dimensionalmodel are increased at an exponential level, which not only imposes agreat computational burden on, for example, the reconstruction andfusion of the three-dimensional model that can be performed by thecloud, but also consumes a large amount of time, thereby affecting thereconstruction and fusion efficiency of the three-dimensional model andreducing the user's experience.

SUMMARY

According to embodiments of the present disclosure, a parallelthree-dimensional reconstruction and fusion method is provided based ona mass driving technique images matched image timing features.

In a first aspect of the present disclosure, there is provided a methodof processing an image, including: acquiring a set of image sequences,the set of image sequences including a plurality of image sequencesubsets divided according to similarity measurement between imagesequences, each image sequence subset including a basic image sequenceand other image sequences, wherein a first similarity measurementcorresponding to the basic image sequence is greater than or equal to afirst similarity measurement corresponding to the other image sequences;creating an original three-dimensional model using the basic imagesequence; and creating a final three-dimensional model using the otherimage sequences based on the original three-dimensional model.

In a second aspect of the present disclosure, there is provided anapparatus for processing an image, including: an image sequence setacquisition module configured to acquire an image sequence set, theimage sequence set including a plurality of image sequence subsetsdivided according to similarity measurements between image sequences,each image sequence subset including a basic image sequence and anotherimage sequence, wherein a first similarity measurement corresponding tothe basic image sequence is greater than or equal to a first similaritymeasurements corresponding to the other image sequences; an originalthree-dimensional model modeling module configured to create an originalthree-dimensional model using the basic image sequence; and a finalthree-dimensional model modeling module configured to create a finalthree-dimensional model using the other image sequences based on theoriginal three-dimensional model.

In a third aspect of the present disclosure, there is provided anelectronic device including at least one processor; and a memory incommunication connection with the at least one processor; wherein thememory stores instructions executable by the at least one processor toenable the at least one processor to implement the method according tothe first aspect of the present disclosure. In a fourth aspect of thepresent disclosure, there is provided a non-transitory computer readablestorage medium having stored thereon computer instructions for causingthe computer to implement a method according to the first aspect of thepresent disclosure.

It is to be understood that what is described in the Summary does notintend to limit the critical or important features of the embodiments ofthe disclosure, nor does intend to limit the scope of the disclosure.Other features of the present disclosure will become readily apparentfrom the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features, and advantages of the presentdisclosure will become more apparent from the more detailed descriptionof exemplary embodiments of the disclosure, taken in conjunction withthe accompanying drawings, in which same reference numerals refergenerally to the same components in the exemplary embodiments of thedisclosure. It is to be understood that the drawings are for a betterunderstanding of the present disclosure and are not to be construed aslimiting the disclosure, where:

FIG. 1 illustrates a schematic diagram of an image processingenvironment 100 in which the method of processing an image according toexemplary embodiments of the present disclosure may be implemented;

FIG. 2 illustrates a flowchart of a method 200 for processing an imageaccording to an embodiment of the present disclosure;

FIG. 3 illustrates a schematic diagram of an interrelationship 300 of animage sequence according to an embodiment of the present disclosure;

FIG. 4 illustrates a schematic diagram of an correlation relationshipand segments 400 of a basic image sequence according to an embodiment ofthe present disclosure;

FIG. 5 illustrates a schematic diagram of an incremental modelingprocess 500 utilizing a basic image sequence according to an embodimentof the present disclosure;

FIG. 6 illustrates a schematic diagram of an original three-dimensionalmodel 600 according to an embodiment of the present disclosure;

FIG. 7 illustrates a schematic diagram of a process 700 for creating afinal three-dimensional model using an original three-dimensional modelaccording to an embodiment of the present disclosure;

FIG. 8 illustrates a schematic block diagram of an image processingapparatus 800 according to an embodiment of the present disclosure; and

FIG. 9 illustrates a schematic block diagram of an electronic device 900according to an embodiment of the present disclosure.

In the various drawings, the same or corresponding reference numeralsindicate the same or corresponding features.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure will be described in more detailbelow with reference to the accompanying drawings. While embodiments ofthe disclosure are shown in the drawings, it is to be understood thatthe disclosure may be implemented in various forms and should not belimited to the embodiments set forth herein. Instead, these embodimentsare provided so that this disclosure will be thorough and complete, andwill fully convey the scope of this disclosure to those skilled in theart.

The term “comprising” and variations thereof, as used herein, means anopen-ended, i.e., “including, but not limited to”. Unless specificallystated, the term “or” means “and/or”. The term “based on” means “basedat least partly on”. The terms “one exemplary embodiment” and “oneembodiment” means “at least one exemplary embodiment”. The term “anotherembodiment” means “at least one further embodiment”. The terms “first”,“second” or the like may refer to different or same objects. Otherexplicit and implicit definitions may also be included below.

As described above in the background, the use of conventionalthree-dimensional model reconstruction and fusion methods involves alarge amount of computation and occupies a large amount of systemresources, thereby affecting the validity and feasibility of thethree-dimensional modeling, reducing the user's experience or evenresulting in a failure to meet user's requirements.

To at least partially address one or more of the above and otherpotential problems, embodiments of the present disclosure propose amethod of processing an image for three-dimensional modeling. In thesolution described in the present disclosure, the matching oftwo-dimensional images and the similarity calculation are used forthree-dimensional fusion and reconstruction, and data are optimized andgrouped before the three-dimensional reconstruction, and thus it avoidsusing massive data to fuse the three-dimensional model, which otherwiseoccupies system resources and high time complexity. Meanwhile, in thesolution described in the present disclosure, the mass image sequencesfrom the automobile data recorder are grouped in accordance with thevideo correlation graphic based on two-dimensional image similarity, sothat the calculation amount of the three-dimensional reconstruction isconverted from the original exponential growth to the linear growth, andthe success rate of modeling can be improved by creating the models inparallel.

With the technology according to the present disclosure, it is possibleto improve the efficiency of creating the three-dimensional model andreduce the overhead of creating the three-dimensional model.

FIG. 1 illustrates a schematic diagram of an image processingenvironment 100 in which the method of processing an image in exemplaryembodiments of the present disclosure may be implemented. As shown inFIG. 1 , the image processing environment 100 includes a computingdevice 120, an image sequence set 110 as input data of the computingdevice 120, and a three-dimensional model 130 as output data of thecomputing device 120. It should be noted that the image processingenvironment 100 is extensible and may include more sets of imagesequences 110 as input data, more three-dimensional model 130 as outputdata, or more computing devices 120 to support more efficient parallelcomputing for the image sequence set 110. For purposes of simplifyingthe diagram, only one image sequence set 110, one computing device 120,and one three-dimensional model 130 are shown in FIG. 1 .

In the description of embodiments of the present disclosure, a set ofimage sequences refers to a set of one or more image sequences, whereineach image in an image sequence corresponds to a frame of imagesacquired by an acquiring images device. According to an embodiment ofthe present disclosure, an image sequence is acquired by an acquiringimages device installed or included on a vehicle or other device foracquiring a driving record image, wherein each acquired frame of imageincludes an ambient environment to which the acquiring images device isfacing and other objects such as a person, a bicycle, a pet, and thelike in the environment. In a process of acquiring images, an acquiringimages device may acquire a frame of an image at time intervals or atdistance intervals, and the images sequentially acquired throughout theacquiring images process form an image sequence for this process ofacquiring images, wherein the image sequence may be embodied in the formof a video.

In the description of embodiments of the present disclosure,three-dimensional model reconstruction and fusion of a set of imagesequences refers to a process of obtaining a three-dimensional model byusing the set of image sequences to model. According to an embodiment ofthe present disclosure, the obtained three-dimensional model may includea three-dimensional point cloud of the surrounding environment of, forexample, a parking lot or a residential road, to which the set of imagesequences relate, as well as the content and related information of eachacquired image.

In the description of embodiments of the present disclosure,three-dimensional model reconstruction and fusion may be a sequentiallinear process, i.e., a modeling process is accomplished by continuallyadding new images to participate in modeling, where the added images mayreach hundreds of thousands or more.

FIG. 2 illustrates a flow diagram of a method 200 for processing animage according to an embodiment of the present disclosure. Inparticular, the method 200 may be performed by the computing device 120.It should be understood that the method 200 may also include additionaloperations not shown and/or omit some of the shown operations, and thescope of the present disclosure is not limited in this respect.

At block 202, the computing device 120 acquires a set of image sequences110. According to an embodiment of the present disclosure, the imagesequence set 110 includes a plurality of image sequence subsets dividedaccording to similarity measurement between image sequences, whereineach image sequence subset includes a basic image sequence and otherimage sequences, and a first similarity measurement corresponding to thebasic image sequence is greater than or equal to the first similaritymeasurement corresponding to the other image sequences.

The set of image sequences 110 is further described below in connectionwith FIG. 3 . FIG. 3 illustrates a schematic diagram of aninterrelationship 300 of the image sequences according to an embodimentof the present disclosure. A total of 13 image sequences are shown inFIG. 3 , i.e., the image sequences 301, 302, 303, 304, 305, 306, 307,308, 309, 310, 311, 312, and 313, which together constitute an exampleof the image sequence set 110. In FIG. 3 , a plurality of image sequencesubsets are shown in circles of dashed line 320, 330, 340, 350, and 360,wherein the image sequence subset 320 includes the image sequences 303,301, and 313, the image sequence subset 330 includes the image sequences302, 303, 304, and 305, the image sequence subset 340 includes the imagesequences 301, 306, and 309, the image sequence subset 350 includes theimage sequences 310, 311, 312, and 313, and the image sequence subset360 includes the image sequence 301, 307, and 308.

In the image sequence shown in FIG. 3 , the image sequences 301, 303,and 313 are basic image sequences, which together constitute an imagesequence subset 320. The basic image sequence includes most of thestable elements, and the correlation between them is highest, which isembodied as having a higher similarity measurement, so that thethree-dimensional modeling can be first performed by using the basicimage sequence, and then further three-dimensional modeling can beperformed by using other image sequences. In addition, the basic imagesequences are included in all the image sequence subset 320, 330, 340,350, and 360, so that the other image sequences can be utilized formodeling after three-dimensional model is created with the basic imagesequence, since these other image sequences all have a high degree ofcorrelation with a certain basic image sequence. In addition, the imagesequence subset 320, 330, 340, 350, and 360 may be divided according tosimilarity measurements. According to an exemplary embodiment of thepresent disclosure, a second similarity measurement corresponding to animage sequence subset in each image sequence subset is greater than orequal to a second similarity measurement corresponding to an imagesequence subset in other subsets of image sequences. According to anembodiment of the present disclosure, the first similarity measurementis greater than or equal to the second similarity measurement.

At block 204, the computing device 120 uses the basic image sequence tocreate an original three-dimensional model. Taking the example in FIG. 3, the computing device 120 uses the image sequences 301, 303, and 313 asbasic image sequences to create an original three-dimensional model.

According to an exemplary embodiment of the present disclosure, when thecomputing device 120 uses the image sequences 301, 303, and 313 as thebasic image sequences to create the original three-dimensional model, itis necessary to first align the image sequences 301, 303, and 313 andadditionally segment them.

FIG. 4 illustrates a schematic diagram of an correlation relationshipand segmenting 400 of basic image sequences according to an embodimentof the present disclosure. According to this exemplary embodiment of thepresent disclosure, in conjunction with the example in FIG. 3 , sincethe start positions for acquiring images of the image sequence 301, 303,and 313 may be different, the scenes for these images are alsodifferent. In three-dimensional modeling, it needs to use imagesacquired at the same acquisition positions. Therefore, it is necessaryto find images acquired for the same acquisition location in differentimage sequences, among which there is a higher similarity measurement,and this operation may also be referred to as determining a correlatedimage between different image sequences. It will be appreciated that theimages in each of the image sequences 301, 303, and 313 are all arrangedin the timing order in which they are acquired, so that when the firstimage in the each sequence is aligned, the subsequent images thereinhave a greater probability of being also aligned, at which point theprocess of aligning the subsequent images by determining a similaritymeasurement may be omitted.

According to an exemplary embodiment of the present disclosure, thecomputing device 120 determines a correlated image for a basic imagesequence between basic image sequences according to at least one of:acquisition locations of the images and a third measurement ofsimilarity between the images.

As shown in FIG. 4 , the three image sequences 303, 301, and 313 arestaggered, i.e. aligned, so that the images at the same location areacquired at the same acquisition location and have a higher similaritymeasurement. According to an exemplary embodiment of the presentdisclosure, different image sequences may have different numbers ofimages, and thus the total lengths of the three image sequences 303,301, and 313 are not the same.

After the computing device 120 determines a correlated image for thebasic image sequence between the basic image sequences, i.e., afteraligning the basic image sequences, a three-dimensional model may becreated using the correlated image for the basic image sequences.

According to an exemplary embodiment of the present disclosure, when thelength of the basic image sequence is long, directly aligning the imagesequences to create the three-dimensional model may involve a largeamount of computation and occupy a large amount of system resources.Therefore, in order to further reduce the computation amount and thesystem resource occupation, the basic image sequence can be segmented.The criteria for segmentation may be the computing power of an availablecomputer or computing thread. The greater the computational power of theavailable computer or computing thread, the less the number of segmentsmay be.

As shown in FIG. 4 , the basic image sequence 303 is divided into fivesegments, namely, segments 303-1, 303-2, 303-3, 303-4, and 303-5; thebasic image sequence 301 is divided into four segments, namely, segments301-2, 301-3, 301-4, and 301-5; and the basic image sequence 313 isdivided into four segments, namely, segments 313-1, 313-2, 313-3, and313-4. As shown in FIG. 4 , the segments 303-1 and 313-1 are aligned,the segments 303-2, 301-2 and 313-2 are aligned, the segments 303-3,301-3 and 313-3 are aligned, the segments 303-4, 301-4 and 313-4 arealigned, and the segments 303-5 and 301-5 are aligned.

According to an exemplary embodiment of the present disclosure, afterthe computing device 120 has segmented the basic image sequences, anoriginal basic segmented three-dimensional model may be created for thefirst segment.

FIG. 5 illustrates a schematic diagram of an incremental modelingprocess 500 utilizing a basic image sequence according to an embodimentof the present disclosure. As shown in FIG. 5 , after modeling with thesegments 303-1 and 313-1 in FIG. 4 , an original basic segmentedthree-dimensional model 501 is obtained. As shown in FIG. 4 , since thesegments 303-2 and 313-2 are segments immediately after segments 303-1and 313-1, the incremental modeling may be performed using the segments303-2 and 313-2 and the corresponding segments 301-2 based on theoriginal basic segmented three-dimensional model 501 to obtainincremental basic segmented three-dimensional model 502 corresponding tosegments 303-2, 313-2 and 301-2. Then, the incremental basic segmentedthree-dimensional model 502 may be used as an original basic segmentedthree-dimensional model, incremental modeling may be performed usingfurther immediate segments 303-3, 313-3, and 301-3 based on the basicsegmented three-dimensional model 502 only, thereby obtaining anincremental basic segmented three-dimensional model 503, and theincremental basic segmented three-dimensional models 504 and 505 byanalogy.

The computing device 120 may then fuse the created original basicsegmented three-dimensional model 501 with the subsequently createdincremental basic segmented three-dimensional models 502, 503, 504, and505 to obtain an original three-dimensional model.

Referring to FIG. 6 , a schematic diagram of an originalthree-dimensional model 600 according to an embodiment of the presentdisclosure is shown. As shown in FIG. 6 , the original three-dimensionalmodel 600 includes basic segmented three-dimensional models 501, 502,503, 504, and 505.

According to an exemplary embodiment of the present disclosure, theoriginal three-dimensional model may be created using the four pairs ofbasic segmented three-dimensional models shown in FIG. 5 as a whole,i.e., using the basic segmented three-dimensional models 501 and 502,the basic segmented three-dimensional models 502 and 503, the basicsegmented three-dimensional models 503 and 504, and the basic segmentedthree-dimensional models 504 and 505. In this case, since each pair ofbasic segmented three-dimensional models has a portion that overlapswith the previous or subsequent pair of basic segmentedthree-dimensional models, the creation of the original three-dimensionalmodel may be more easily achieved.

According to an exemplary embodiment of the present disclosure, aplurality of incremental basic segmented three-dimensional models may becreated in parallel. For example, as shown in FIG. 4 , the originalbasic segmented three-dimensional model 503 may be first created usingthe corresponding segments 303-3, 301-3, and 313-3, and then incrementalmodeling may be performed in parallel based on the original basicsegmented three-dimensional model 503 by using two computers ordifferent threads of the same computer, utilizing the segments 303-2,313-2, and 301-2, or segments 303-4, 313-4, and 301-4, respectively, toobtain incremental original basic segmented three-dimensional models 502and 504 in parallel.

Returning to FIG. 2 , at 206, the computing device 120 creates a finalthree-dimensional model using other image sequences based on theoriginal three-dimensional model.

According to an embodiment of the present disclosure, if the originalthree-dimensional model does not involve segmentation, and all imagesequences are naturally aligned, the computing device 120 may create aplurality of intermediate three-dimensional models based on the originalthree-dimensional model using other image sequences in a plurality ofimage sequence subsets, respectively, wherein each intermediatethree-dimensional model corresponds to one subset of image sequences.Then, the computing device 120 fuses the plurality of intermediatethree-dimensional models to obtain a final three-dimensional model. Itwill be appreciated that since the process of creating the intermediatethree-dimensional model based on the original three-dimensional model isan incremental modeling, the original three-dimensional model portion ofthe obtained intermediate three-dimensional models are the same, so thatthey may be more easily fused to obtain the final three-dimensionalmodel. According to embodiments of the present disclosure, the processesof creating the intermediate three-dimensional model may be performed inparallel using a plurality of different computers or different computingthreads, so that the speed at which the final three-dimensional model isobtained may be increased.

According to an embodiment of the present disclosure, if the originalthree-dimensional model does not involve segmentation, and all imagesequences are not naturally aligned, it is also necessary to align allimage sequences. For example, the computing device 120 may determine acorrelated image between image sequences in an image sequence set basedon at least one of acquisition locations of the images, and a thirdmeasurement of similarity between the images, wherein the correlatedimage includes a correlated image for a basic image sequence and acorrelated image for other image sequences. This alignment process issimilar to that described above with respect to FIG. 4 and will not berepeated here.

According to an embodiment of the present disclosure, if the originalthree-dimensional model consists of a plurality of basic segmentedthree-dimensional models, the original three-dimensional model 600 shownin FIG. 6 as example consists of the basic segmented three-dimensionalmodels 501, 502, 503, 504, and 505, and all image sequences are notnaturally aligned. The computing device 120 will first align all imagesequences as described above, then, the computing device 120 willsegment the other image sequences according to the segmentation of thebasic image sequence.

FIG. 7 illustrates a schematic diagram of a process 700 for creating afinal three-dimensional model using an original three-dimensional modelaccording to an embodiment of the present disclosure. The four dashedline boxes 701, 702, 703 and 704 in FIG. 7 represent four differentcomputing devices, respectively, which may also be servers, and whichmay be cloud computing devices. In each computing device, the process ofincremental modeling is performed based on an original three-dimensionalmodel consisting of the basic segmented three-dimensional models 501,502, 503, 504, and 505, the purpose of distinguishing differentcomputing devices is to illustrate that these incremental modelingprocesses may be performed in parallel by these computing devices.

According to an embodiment of the present disclosure, all other imagesequences are aligned with the basic image sequences 303, 301, and 313and correspondingly divided into segments, wherein the other imagesequence 302 is divided into segments 302-1, 302-2, 302-3, 302-4, and302-5, the other image sequence 304 is divided into segments 304-2,304-3, 304-4, and 304-5, the other image sequence 305 is divided intosegments 305-1, 305-2, 305-3, and 305-4, the other image sequence 306 isdivided into segments 306-1, 306-2, 306-3, 306-4, and 306-5, the otherimage sequence 309 is divided into segments 309-2, 309-3, 309-4, and309-5, the other image sequence 308 is divided into segments 308-2,308-3, 308-4, and 308-5, the other image sequence 307 is divided intosegments 307-1, 307-2, 307-3, and 307-4, the other image sequence 310 isdivided into segments 310-1, 312-2, 310-3, 310-4, and 310-5, and theother image sequence 311 is divided into segments 311-2, 311-3, 311-4,and 312-5, and other image sequence 312 is divided into segments 312-1,312-2, 312-3, and 312-4.

Then, the four computing devices 701, 702, 703, and 704 respectively usesegments divided from the basic image sequences 302, 304, 305, 306, 309,308, 307, 310, 311, and 312 to perform the incremental modeling, basedon the original three-dimensional model consisting of the basicsegmented three-dimensional models 501, 502, 503, 504, and 505. Whereinthe different segments connected by the line segments indicate thecorresponding relationship of the segments with the basic segmentedthree-dimensional model, and the processes of incremental modeling ofthe corresponding entire segments by one basic segmentedthree-dimensional model may be performed in parallel with the processesof incremental modeling of the corresponding entire segments by otherbasic segmented three-dimensional models, e.g., by different threads inthe computing devices 701, 702, 703, and 704.

The above describes an image processing environment 100 in which themethod of processing an image in certain exemplary embodiments of thepresent disclosure may be implemented, a method for processing an image200 according to an embodiment of the present disclosure, aninterrelationship 300 of image sequences according to an embodiment ofthe present disclosure, a correlation and segment 400 of a basic imagesequence according to an embodiment of the present disclosure, anincremental modeling process 500 using the basic image sequenceaccording to an embodiment of the present disclosure, an originalthree-dimensional model 600 according to an embodiment of the presentdisclosure, and related content of a process 700 using the originalthree-dimensional model to create a final three-dimensional modelaccording to an embodiment of the present disclosure, with reference toFIGS. 1 to 7 . It is to be understood that the foregoing descriptionintends to provide a better illustration of what is recited in thepresent disclosure, and does not intend to be limiting in any way.

It is to be understood that the number and magnitude of the variouselements illustrated in the various figures of the present disclosureare by way of example only and does not intend to limit the scope ofprotection of the disclosure. The above numbers and sizes may bearbitrarily set as desired without affecting the normal implementationof the embodiments of the present disclosure.

Details of a method of processing an image according to an embodiment ofthe present disclosure have been described above with reference to FIGS.1-7 . Hereinafter, various modules in an apparatus for processing animage will be described with reference to FIG. 8 .

FIG. 8 is a schematic block diagram of an apparatus 800 for processingan image according to an embodiment of the present disclosure. As shownin FIG. 8 , the apparatus 800 may include an image sequence setacquisition module 810 configured to acquire an image sequence setincluding a plurality of image sequence subsets divided according to asimilarity measurement between image sequences, wherein each imagesequence subset includes a basic image sequence and other imagesequences, and wherein a first similarity measurement corresponding tothe basic image sequence is greater than or equal to a first similaritymeasurement corresponding to the other image sequences. The apparatus800 may further include an original three-dimensional model modelingmodule 820 configured to create an original three-dimensional modelusing the basic image sequence and a final three-dimensional modelmodeling module 830 configured to create a final three-dimensional modelusing the other image sequences based on the original three-dimensionalmodel.

In some embodiments, wherein the second similarity measurementcorresponding to the image sequence subset in each image sequence subsets is greater than or equal to the second similarity measurementcorresponding to image sequence subset in other image sequence subset.

In certain embodiments, wherein the original three-dimensional modelmodeling module 820 includes a first correlated image determining module(not shown) configured to determine a correlated image for a basic imagesequence between the basic image sequences based on at least one ofacquisition locations of the images, and a third measurement ofsimilarity between the images; and a first original three-dimensionalmodel modeling module (not shown) configured to create the originalthree-dimensional model using the correlated image for the basic imagesequence.

In certain embodiments, wherein the original three-dimensional modelmodeling module 820 includes a basic segmentation division module (notshown) configured to divide the correlated image for the basic imagesequence into a plurality of corresponding basic segments in an order ofacquiring the images; an original basic segmented three-dimensionalmodel modeling module (not shown) configured to create an original basicsegmented three-dimensional model using one corresponding basic segmentof the plurality of corresponding basic segments; an incremental basicsegmented three-dimensional model modeling module (not shown) configuredto create an incremental basic segmented three-dimensional model using acorresponding basic segment adjacent to the one corresponding basicsegment based on the created original basic segmented three-dimensionalmodel; and a second original three-dimensional model modeling module(not shown) configured to fuse the original basic segmentedthree-dimensional model and the incremental basic segmentedthree-dimensional model to obtain the original three-dimensional model.

In certain embodiments, the incremental basic segmentedthree-dimensional model modeling module is configured to create aplurality of the incremental basic segmented three-dimensional models inparallel.

In some embodiments, the final three-dimensional model modeling module830 includes a first intermediate three-dimensional model modelingmodule (not shown) configured to create a plurality of intermediatethree-dimensional models based on the original three-dimensional modelusing the other image sequences in the plurality of image sequencesubsets, respectively; and a first final three-dimensional modelmodeling module (not shown) configured to fuse the plurality ofintermediate three-dimensional models to obtain the finalthree-dimensional model.

In certain embodiments, the intermediate three-dimensional modelmodeling module is configured to create a plurality of the intermediatethree-dimensional models in parallel.

In some embodiments, the final three-dimensional model modeling module830 includes a second correlated image determining module (not shown)configured to determine an correlated image between image sequences inthe set of image sequences based on at least one of acquisitionlocations of the images, and a third measurement of similarity betweenthe images, wherein the correlated image includes a correlated image forthe basic image sequence and a correlated image for the other imagesequences; and a second final three-dimensional model modeling module(not shown) configured to create the final three-dimensional model usinga correlated image for the other image sequence.

In some embodiments, the final three-dimensional model modeling module830 includes a second correlated image determining module (not shown)configured to determine an correlated image between image sequences inthe set of image sequences based on at least one of acquisitionlocations of the images, and a third measurement of similarity betweenthe images, wherein the correlated image includes a correlated image forthe basic image sequence and a correlated image for the other imagesequences; a segment dividing module (not shown) configured to dividethe correlated image into a plurality of corresponding segments in anacquiring images order, the plurality of corresponding segmentsincluding the plurality of corresponding basic segments and a pluralityof corresponding other segments divided from the other image sequences;a first intermediate three-dimensional model modeling module (not shown)configured to create a plurality of intermediate three-dimensionalmodels based on the original three-dimensional model using the othersegments corresponding to the original basic segmented three-dimensionalmodel and the incremental basic segmented three-dimensional model; and afirst final three-dimensional model modeling module (not shown)configured to fuse the plurality of intermediate three-dimensionalmodels to obtain the final three-dimensional model.

In certain embodiments, the first intermediate three-dimensional modelmodeling module is configured to create a plurality of the intermediatethree-dimensional models in parallel.

According to an embodiment of the present disclosure, the presentdisclosure also provides an electronic device and a readable storagemedium.

From the above description with reference to FIGS. 1 to 8 , thetechnical solution according to an embodiment of the present disclosurehas a number of advantages over conventional solutions. For example, byusing the technical solution, the video sequence correlation mapping maybe fully utilized, and the data fused with the three-dimensionalreconstruction model may be reasonably divided, so that thecomputational complexity of mass three-dimensional fusion modeling onthe cloud end is greatly reduced, and the process is suitable forperforming distributed parallel processing, and the computing capabilityof the company cluster is fully utilized, thereby greatly improving thecomputational efficiency.

FIG. 9 shows a schematic block diagram of an electronic device 900according to an embodiment of the present disclosure. For example, thecomputing device 120 shown in FIG. 1 and the apparatus 800 forprocessing an image shown in FIG. 8 may be implemented by an electronicdevice 900. Electronic device 900 is intended to represent various formsof digital computers, such as laptop computers, desktop computers,worktables, personal digital assistants, servers, blade servers,mainframe computers, and other suitable computers. Electronic device 900may also represent various forms of mobile devices, such as personaldigital processing, cellular telephones, smart phones, wearable devices,and other similar computing devices. The components shown herein, theirconnections and relationships, and their functions are by way of exampleonly and are not intended to limit the implementation of the disclosuredescribed and/or claimed herein.

As shown in FIG. 9 , the electronic device 900 includes one or moreprocessors 901, a memory 902, and an interface for connectingcomponents, including a high speed interface and a low speed interface.The various components are interconnected by different buses and may bemounted on a common motherboard or otherwise as desired. The processormay process instructions executed within the electronic device 900,including instructions stored in or on a memory to display graphicalinformation of the GUI on an external input/output device, such as adisplay device coupled to an interface. In other embodiments, multipleprocessors and/or multiple buses may be used with multiple memories, ifdesired. Similarly, a plurality of electronic devices 900 may beconnected, each providing a portion of the necessary operations (e.g.,as a server array, a set of blade servers, or a multiprocessor system).One processor 901 is exemplified in FIG. 9 .

Memory 902 is a non-transitory computer readable storage medium providedby the present disclosure. Wherein the memory stores instructionsexecutable by at least one processor to cause the at least one processorto perform the method of processing an image provided by the presentdisclosure. The non-transient computer-readable storage medium of thepresent disclosure stores computer instructions for causing a computerto perform the method of processing an image provided by the presentdisclosure.

The memory 902, as a non-transitory computer readable storage medium,may be used to store non-transitory software programs, non-transitorycomputer executable programs, and modules, such as programinstructions/modules corresponding to the method of processing images inembodiments of the present disclosure (e.g., the image sequence setacquisition module 810, the original three-dimensional model modelingmodule 820, and the final three-dimensional model modeling module 830shown in FIG. 8 ). The processor 901 executes various functionalapplications and data processing of the server by running non-transitorysoftware programs, instructions, and modules stored in the memory 902,that is, implements the method of processing an image in theabove-described method embodiment.

The memory 902 may include a storage program area and a storage dataarea, wherein the storage program area may store an operating system, anapplication program required by at least one function; the storage dataarea may store data or the like created according to the use of theelectronic device 900. In addition, memory 902 may include high speedrandom access memory, and may also include non-transitory memory, suchas at least one magnetic disk storage device, flash memory device, orother non-transitory solid state storage device. In some embodiments,memory 902 may optionally include remotely disposed memory relative toprocessor 901, which may be connected to electronic device 900 via anetwork. Examples of such networks include, but are not limited to, theInternet, enterprise intranets, local area networks, mobilecommunication networks, and combinations thereof.

The electronic device 900 may also include an input device 903 and anoutput device 904. The processor 901, the memory 902, the input device903, and the output device 904 may be connected via a bus or otherwise,as illustrated in FIG. 9 .

The input device 903 may receive input digit or character informationand generate key signal input related to user settings and functionalcontrol of the electronic device 900, such as a touch screen, a keypad,a mouse, a track pad, a touch pad, a pointer bar, one or more mousebuttons, a trackball, a joystick, or the like. The output device 904 mayinclude a display device, an auxiliary lighting device (e.g., an LED), atactile feedback device (e.g., a vibration motor), and the like. Thedisplay device may include, but is not limited to, a liquid crystaldisplay (LCD), a light emitting diode (LED) display, and a plasmadisplay. In some embodiments, the display device may be a touch screen.

The various embodiments of the systems and techniques described hereinmay be implemented in digital electronic circuit systems, integratedcircuit systems, application specific ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various embodiments may include beingimplemented in one or more computer programs that may execute and/orinterpret on a programmable system including at least one programmableprocessor, which may be a dedicated or general purpose programmableprocessor, may receive data and instructions from a memory system, atleast one input device, and at least one output device, and transmit thedata and instructions to the memory system, the at least one inputdevice, and the at least one output device.

These computing programs (also referred to as programs, software,software applications, or code) include machine instructions of aprogrammable processor and may be implemented in high-level proceduresand/or object-oriented programming languages, and/or assembly/machinelanguages. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,device, and/or means (e.g., magnetic disk, optical disk, memory,programmable logic device (PLD)) for providing machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as machine-readable signals.The term “machine readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques describedherein may be implemented on a computer having a display device (e.g., aCRT (cathode ray tube) or LCD (liquid crystal display) monitor) fordisplaying information to the user; and a keyboard and a pointing device(e.g., a mouse or a trackball) through which a user can provide input toa computer. Other types of devices may also be used to provideinteraction with a user; for example, the feedback provided to the usermay be any form of sensory feedback (e.g., visual feedback, auditoryfeedback, or tactile feedback); and input from the user may be receivedin any form, including acoustic input, speech input, or tactile input.

The systems and techniques described herein may be implemented in acomputing system including a background component (e.g., as a dataserver), or a computing system including a middleware component (e.g.,an application server), or a computing system including a front-endcomponent (e.g., a user computer having a graphical user interface or aweb browser through which a user may interact with embodiments of thesystems and techniques described herein), or a computing systemincluding any combination of such background component, middlewarecomponent, or front-end component. The components of the system may beinterconnected by any form or medium of digital data communication(e.g., a communication network). Examples of communication networksinclude a local area network (LAN), a wide area network (WAN), and theInternet.

The computer system may include a client and a server. The client andserver are typically remote from each other and typically interactthrough a communication network. The relationship between the client andthe server is generated by a computer program running on thecorresponding computer and having a client-server relationship with eachother.

According to the technical solution of the embodiment of the presentdisclosure, by dividing a set of image sequences including an imagesequence set into one or more image sequence subset according to thesimilarity between the image sequences, and then determining in eachimage sequence subset the correlation degree between an image in oneimage sequence and an image in other image sequence in the imagesequence subset, an image having a high correlation degree can beeffectively determined, so that subsequent time-consuming andresource-consuming feature matching calculation can be performed onlyfor the determined image with high correlation degree, thereby reducingthe calculation amount and improving the calculation efficiency.Furthermore, since the image sequence set is divided into one or moreimage sequence subset, different image sequence subset may be assignedto a different computing device to perform parallel feature matchingcalculations. In this way, it is possible to make full use of thecalculation resources, further reduce the calculation time, and improvethe calculation efficiency.

It is to be understood that reordering, adding or deleting of the stepsmay be performed using the various forms shown above. For example, thesteps described in the present disclosure may be performed in parallelor sequentially or in a different order, so long as the desired resultsof the technical solution disclosed in the present disclosure can berealized, and no limitation is imposed herein.

The foregoing detailed description is not intended to limit the scope ofthe present disclosure. It will be appreciated by those skilled in theart that various modifications, combinations, sub-combinations, andsubstitutions may be made depending on design requirements and otherfactors. Any modifications, equivalents, and improvements that fallwithin the spirit and principles of the disclosure are intended to beincluded within the scope of protection of the disclosure.

What is claimed is:
 1. A method for processing an image, comprising:acquiring a set of image sequences, the set of image sequencescomprising a plurality of image sequence subsets divided according tosimilarity measurements between image sequences, each image sequencesubset comprising a basic image sequence and other image sequence,wherein a first similarity measurement corresponding to the basic imagesequence is greater than a first similarity measurement corresponding tothe other image sequence, wherein a correlation between images in thebasic image sequence is highest in the set of image sequences, and thehighest correlation is embodied as having a higher similaritymeasurement; creating an original three-dimensional model using thebasic image sequence in which images have the highest correlation; andcreating a final three-dimensional model using the other image sequencein which images have a lower correlation based on the originalthree-dimensional model.
 2. The method of claim 1, wherein a secondsimilarity measure corresponding to an image sequence subset in eachimage sequence subset is greater than or equal to a second similaritymeasure corresponding to an image sequence subset in other imagesequence subsets.
 3. The method of claim 1, wherein creating an originalthree-dimensional model comprises: determining a correlated image foreach basic image sequence between the basic image sequences according toat least one of: acquisition positions of images, and a third similaritymeasure between the images; and creating the original three-dimensionalmodel using the correlated image for the basic image sequence.
 4. Themethod of claim 3, wherein creating the original three-dimensional modelcomprises: dividing the correlated images into a plurality ofcorresponding basic segments according to an order of acquiring theimages; creating an original basic segmented three-dimensional modelusing one corresponding basic segment of the corresponding basicsegments; creating an incremental basic segmented three-dimensionalmodel using a corresponding basic segment adjacent to the onecorresponding basic segment, based on the created original basicsegmented three-dimensional model; and fusing the created original basicsegmented three-dimensional model and the created incremental basicsegmented three-dimensional model to obtain the originalthree-dimensional model.
 5. The method of claim 4, wherein a pluralityof the incremental basic segmented three-dimensional models are createdin parallel.
 6. The method of claim 4, wherein creating the finalthree-dimensional model comprises: determining the correlated imagesbetween image sequences in the set of image sequences according to atleast one of: acquisition positions of the images, and the thirdsimilarity measure between the images, wherein the correlated imagescomprise a correlated image for the basic image sequence and acorrelated image for the other image sequence; dividing the correlatedimages into a plurality of corresponding segments according to an orderof acquiring images, the plurality of corresponding segments comprisinga plurality of corresponding basic segments and a plurality ofcorresponding other segments divided from the other image sequences;creating a plurality of intermediate three-dimensional models based onthe original three-dimensional model using the corresponding othersegments corresponding to the original basic segmented three-dimensionalmodel and the incremental basic segmented three-dimensional model; andfusing the plurality of intermediate three-dimensional models to obtainthe final three-dimensional model.
 7. The method of claim 6, wherein aplurality of the intermediate three-dimensional models are created inparallel.
 8. The method of claim 3, wherein creating the finalthree-dimensional model comprises: determining the correlated imagesbetween image sequences in the acquired set of image sequences accordingto at least one of: acquisition positions of the images, and a thirdsimilarity measure between the images, wherein the correlated imagescomprise a correlated image for the basic image sequence and acorrelated image for the other image sequence; and creating the finalthree-dimensional model using the correlated image for the other imagesequence.
 9. The method of claim 1, wherein creating the finalthree-dimensional model comprises: creating a plurality of intermediatethree-dimensional models based on the original three-dimensional modelusing the other image sequence in the plurality of image sequencesubsets, respectively; and fusing the plurality of intermediatethree-dimensional models to obtain the final three-dimensional model.10. The method of claim 9, wherein the plurality of the intermediatethree-dimensional models are created in parallel.
 11. An electronicdevice, comprising: at least one processor; and a memory incommunication connection with the at least one processor; wherein, thememory stores instructions executable by the at least one processor, theinstructions are performed by the at least one processor to enable theat least one processor to perform operations comprising: acquiring a setof image sequences, the set of image sequences comprising a plurality ofimage sequence subsets divided according to similarity measurementsbetween image sequences, each image sequence subset comprising a basicimage sequence and other image sequence, wherein a first similaritymeasurement corresponding to the basic image sequence is greater than afirst similarity measurement corresponding to the other image sequencewherein a correlation between images in the basic image sequence ishighest in the set of image sequences, and the highest correlation isembodied as having a higher similarity measurement; creating an originalthree-dimensional model using the basic image sequence in which imageshave the highest correlation; and creating a final three-dimensionalmodel using the other image sequence in which images have a lowercorrelation based on the original three-dimensional model.
 12. Thedevice of claim 11, wherein a second similarity measure corresponding toan image sequence subset in each image sequence subset is greater thanor equal to a second similarity measure corresponding to an imagesequence subset in other image sequence subsets.
 13. The device of claim11, wherein the creating an original three-dimensional model comprises:determining a correlated image for each basic image sequence between thebasic image sequences according to at least one of: acquisitionpositions of images, and a third similarity measure between the images;and creating the original three-dimensional model using the correlatedimage for the basic image sequence.
 14. The device of claim 13, whereincreating the original three-dimensional model comprises: dividing thecorrelated images into a plurality of corresponding basic segmentsaccording to an order of acquiring the images; creating an originalbasic segmented three-dimensional model using one corresponding basicsegment of the corresponding basic segments; creating an incrementalbasic segmented three-dimensional model using a corresponding basicsegment adjacent to the one corresponding basic segment, based on thecreated original basic segmented three-dimensional model; and fusing thecreated original basic segmented three-dimensional model and the createdincremental basic segmented three-dimensional model to obtain theoriginal three-dimensional model.
 15. The device of claim 14, wherein aplurality of the incremental basic segmented three-dimensional models iscreated in parallel.
 16. The device of claim 14, wherein creating thefinal three-dimensional model comprises: determining the correlatedimages between image sequences in the set of image sequences accordingto at least one of: acquisition positions of the images, and the thirdsimilarity measure between the images, wherein the correlated imagescomprise a correlated image for the basic image sequence and acorrelated image for the other image sequence; dividing the correlatedimages into a plurality of corresponding segments according to an orderof acquiring images, the plurality of corresponding segments comprisinga plurality of corresponding basic segments and a plurality ofcorresponding other segments divided from the other image sequences;creating a plurality of intermediate three-dimensional models based onthe original three-dimensional model using the corresponding othersegments corresponding to the original basic segmented three-dimensionalmodel and the incremental basic segmented three-dimensional model; andfusing the plurality of intermediate three-dimensional models to obtainthe final three-dimensional model.
 17. The device of claim 13, whereincreating the final three-dimensional model comprises: determining thecorrelated images between image sequences in the acquired set of imagesequences according to at least one of: acquisition positions of theimages, and a third similarity measure between the images, wherein thecorrelated images comprise a correlated image for the basic imagesequence and a correlated image for the other image sequence; andcreating the final three-dimensional model using the correlated imagefor the other image sequence.
 18. The device of claim 11, whereincreating the final three-dimensional model comprises: creating aplurality of intermediate three-dimensional models based on the originalthree-dimensional model using the other image sequence in the pluralityof image sequence subsets, respectively; and fusing the plurality ofintermediate three-dimensional models to obtain the finalthree-dimensional model.
 19. The device of claim 18, wherein theplurality of the intermediate three-dimensional models are created inparallel.
 20. A non-transitory computer-readable storage medium storingcomputer instructions for causing a computer to perform operationscomprising: acquiring a set of image sequences, the set of imagesequences comprising a plurality of image sequence subsets dividedaccording to similarity measurements between image sequences, each imagesequence subset comprising a basic image sequence and other imagesequence, wherein a first similarity measurement corresponding to thebasic image sequence is greater than a first similarity measurementcorresponding to the other image sequence, wherein a correlation betweenimages in the basic image sequence is highest in the set of imagesequences, and the highest correlation is embodied as having a highersimilarity measurement; creating an original three-dimensional modelusing the basic image sequence in which images have the highestcorrelation; and creating a final three-dimensional model using theother image sequence in which images have a lower correlation based onthe original three-dimensional model.